Hybrid POS tagging with generalized unknown-word handling

This paper presents POSTAG 1 as a statistical/rule-based hybrid part-of-speech (POS) tagging system with generalized unknown-word handling. The POSTAG integrates morphological analysis with statistical POS disambigua-tion and post rule-based error-correction. The error-correction rules are automatically learned from a tagged corpus and selectively correct standard HMM tagging errors. The morphological analysis is tightly coupled with the generalized unknown-word handling which uses a morpheme-pattern dictionary that encodes general lexical patterns of Korean morphemes. In this way, we can guess the POS's of unknown-words regardless of their numbers and positions in an eojeol. Experiments demonstrate the eeectiveness of our POS tagging methods with generalized unknown-word handling, and the POSTAG will be especially suited to web indexing where most of the indexing terms are unknown words.